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ABSTRACT 

Kaplan and Norton introduced the Balanced Scorecard (BSC) which is based on a systems perspective of the business 
strategy and performance measurement. Many organizations around the world are using the BSC to define, implement 
and manage strategy. Nevertheless there exist studies that identify problems and limitations associated with the 
implementation and use of the BSC. Those studies show in general terms that managers do not understand the BSC as the 
measures and perspectives in use are fairly independent, and do not always mirror the recommended cause-and-effect 
logic included in the systems perspective of the BSC approach. This article addresses the effectiveness of teaching the 
Balanced Scorecard by means of business simulation. An experiment that uses a business simulator is performed for 
testing a set of hypotheses about the influence of simulation on the students’ understanding of the BSC. The simulation 
experience was specifically designed by the authors to promote understanding of the BSC concepts. Student feedback and 
assessment showed that the simulation significantly enhanced the understanding of the BSC concepts related to the 
strategic management and double-loop learning processes and the systems perspective. Results also suggested that 
understanding of those BSC concepts positively influences simulation performance. 
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1. INTRODUCTION 

Kaplan and Norton (1992) introduced the Balanced Scorecard (BSC) with the aim of overcoming strategic 
management limitations of the traditional performance measurement systems. The BSC approach features a 
mix of lead (performance drivers) and lag (outcome measures) indicators, financial and non-financial 
measures. These are categorized as follows: (i) financial, (ii) customer, (hi) process and (iv) learning and 
growth. The BSC tool helps managers monitor actual financial and market performance, evaluate the results 
of short-term processing actions, monitor the intangible development of competencies that will drive future 
financial performance and assess the progress of implementing corporate strategy. To support managers in 
developing a cause-and-effect perspective and to better understand the business system in which they 
participate, Kaplan and Norton (2001) developed the strategy map concept as a complementary tool to the 
BSC approach. The strategy map links the performance indicators in a causal chain (causal diagram) that 
helps managers to translate, test and communicate their understanding of the business system. 

Managers make decisions and learn in the context of feedback loops (Forrester, 1961). In single-loop 
learning, managers compare information about the state of a real system to pre-established goals, perceive 
deviations between desired and actual states, and make the decisions they believe will move the system 
towards the desired state. Single-loop learning does not change the managers’ mental models. A mental 
model is a conceptual representation of the structure of an external system used by people to describe, 
explain and predict a system’s behavior (Craik, 1943, Johnson-Laird, 1983). In double-loop learning, 
information about the business system is not only used to make decisions within the context of existing 
frames, but also feeds back to modify the managers’ mental models (Argyris, 1976). As their mental models 
change, managers define new strategies and policies. 

It is clear that the BSC approach is consistent with the systems and feedback learning perspective of 
business management and performance measurement. Kaplan and Norton (2001) argue that the BSC 
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approach supports double-loop learning that facilitates managerial strategic learning, leading to better 
performance. In a continual process, managers use the balanced scorecard and the strategy map to re-evaluate 
the assumptions used in the previous strategy. They review the assumed cause-and-effect relationships and 
identify new ones. Then they improve their understanding of the business system and they determine a new 
strategy (Kaplan and Norton, 2001). In other words, BSC triggers a process by which managers can make 
explicit improvements to their mental models of the business system. They adapt the company strategy and 
define the new short and middle term objectives by simulating their mental models to infer the future 
behavior of the business system. 

Many organizations around the world are using the BSC approach to define, implement and manage 
strategy. However, according to empirical research into the performance implications of the BSC, the 
positive contributions of the BSC have not been unambiguously confirmed. There exist studies in the field of 
management accounting research that identify problems and limitations associated with the BSC approach. 
The inadequate definition and utilization of the performance indicators has been highlighted as a main 
drawback of the BSC system (Tingle and Schiemann, 1996; Stivers et ak, 1998; Ittner and Larcker, 1998, 
2003; Lipe and Salterio, 2000; Malmi, 2001; Speckbacher et al, 2003). Those studies show in general terms 
that the measures and perspectives in use are fairly independent, and do not always mirror the recommended 
cause-and-effect logic of the BSC approach. 

1.1 The use of Simulation for teaching the BSC 

Business simulators, business games, management simulators, microworlds, learning laboratory, interactive 
learning environments are some examples of terms used to describe computer-based simulations that have 
been proposed by researchers from many fields, e.g., management and decision science, psychology, 
education and computer science, as important tools to support management learning processes (Maier and 
Grobler, 2000). In a broad definition, business simulators for learning purposes comprise virtually everything 
connected with the learning process. An interactive learning environment contains more than a computer 
simulation model. A simulation model is embedded into a learning environment, which may also include 
case descriptions, presentations by a facilitator, modelling tools, background information, source material and 
working instructions (Davidsen and Spector 1997). 

Simulators are promising tools for teaching in business management domain as they expected to help 
students acquire knowledge. Students’ learning processes center around the exploration of the simulation 
model. They gather knowledge through an inquiry learning process as they make a broad analysis of the 
domain, generate hypothesis, experiment, interpret the outcomes, conclude about the validity of the 
hypothesis or form new ideas, and finally reflect on the domain (de Jong and van Joolingen, 2008). 
Additionally, by performing simulated tasks, learners understand through experiential processes how 
concepts are applied and why they are useful, thus enabling new learning to be more easily (Cannon-Bowers 
and Bowers, 2008). Other specific advantages have been pointed out. An often named advantage of computer 
simulations is the “compression of time” as they instantly show the results of decisions a user has made. 
Simulators help subjects leverage their domain-rich knowledge by allowing them to play through simulated 
years, reflect on their actions, modify their mental models, then repeat the process. By compressing time, 
business simulators can accelerate learning by enabling them to conduct many such cycles of action and 
reflection (Bakken et al, 1994). Providing a safe environment is at the same time an important advantage of 
simulators (Sterman 1994), which allows for experiential learning without that stress-related obstacles that 
are met in reality. However, subjects may tend to take more risks than in real-life decisions. Also, computer 
simulations of business systems address objectively certain special management issues and try to abstract 
from details and isolating from confounding factors (Isaacs and Senge, 1994). This abstraction allows a focus 
on the learning of important and specific business themes. 

As mentioned in previous section, several studies show in general terms that managers do not understand 
the fundamental concepts of the BSC approach, in particular those connected with the systems and feedback 
learning perspective. Thus, appropriate teaching and training methodologies must be designed in order to 
improve subjects learning and comprehension of those concepts. The present study addresses the 
effectiveness of teaching the Balanced Scorecard by means of business simulation. An experiment that uses a 
business simulator is performed for testing a set of hypotheses about the influence of simulation on the 
subjects’ understanding of the BSC. The effectiveness and relevance of simulators for learning is intuitively 


214 



lADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2012) 


acknowledged by many researchers. However, there exists scarce research about the effectiveness of 
computer-based simulations to support learning. Thus, this study also aims to be a contribution to the 
research issue of “effectiveness of computer-based simulations to support learning” not been solved so far. 


2. RESEARCH HYPOTHESES 

This research focuses on how business simulation facilitates the teaching of the Balanced Scorecard 
approach, leading to enhanced students’ understanding of its main concepts. The analysis of the subjects’ 
understanding of the BSC concepts considers two components: the performance measurement system and the 
systems and strategic learning perspective. The performance measurement system associated with the BSC 
approach is viewed as a comprehensive structure of performance indicators that combine both financial and 
non-financial measures. By using a BSC performance measurement system, managers access more relevant 
information to interact with the business environment, and they acquire a sharper understanding of the impact 
of their decisions. The systems and strategic learning perspective - The BSC approach is consistent with the 
systems and strategic learning perspective. By using the BSC strategy map tool, managers will benefit from 
more effective double-loop learning as they review the critical cause-and-effect relations through a process 
that externalizes and improves their mental models of the business system. 

It is assumed that by teaching the BSC through business simulation, students will benefit from more 
effective understanding of the BSC performance measurement system concepts. Hypothesis 1: The use of 
simulation improves the level of understanding of the BSC performance measurement system. 

It is assumed that by teaching the BSC through business simulation, students will benefit from more 
effective understanding of the systems and strategic learning perspective of the BSC approach. Hypothesis 2: 
The use of simulation improves the level of understanding of the BSC as a strategic learning system. 
Hypothesis 3: The use of simulation improves the level of perceived relevance of the strategic learning 
system concepts. 

This study also assumes that students perform better the business simulation task if they reach both higher 
level of understanding of the BSC performance measurement system and higher level of understanding of the 
systems and strategic learning perspective of the BSC approach. Hypothesis 4: The level of understanding of 
the BSC performance measurement system is positively correlated with Performance. Hypothesis 5: The 
level of understanding of the BSC as a strategic learning system is positively correlated with Performance. 
Hypothesis 6: The level of perceived relevance of the strategic learning system concepts is positively 
correlated with Performance. 


3. METHOD 

The hypotheses defined in this research and presented in the previous section were tested with a simulation- 
based experiment. This section presents an overview of the simulator, describes the subjects and the 
experiment conditions, and overviews the research variables. 

The business simulator was built by incorporating the same system dynamics model that had been used in 
previous research (Capelo and Dias, 2009). The participants run a realistic simulator of a wireless 
telecommunications firm by making critical decisions every six months for a simulation period of seven 
years. The participant objective was to develop critical and interrelated resources at appropriate rates and 
levels in order to gain and retain customers, operate efficiently, and maximize value creation. To succeed in 
this simulation task, participants had to identify and understand the cause-and-effect relationships among 
critical variables. The simulator provides an interface that represents a balanced scorecard that includes a set 
of leading and lagging, financial and non-financial indicators that are graphically separated into four sections 
related to the four perspectives associated with the BSC approach. The simulation task also involves drawing 
and reviewing a strategy map. The participants produced and reviewed a strategy map linking the critical 
concepts as were found in the simulation model. This strategy map represents the participants’ understanding 
of the structure of the simulated business system. 
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The subjects consisted of 74 undergraduate students. All the participants were familiar with basic BSC 
concepts as they attended previous classes on the BSC approach. The participants had no experience with the 
simulator and they also had no prior specific knowledge about wireless telecommunications businesses. 

All participants were given a full experimental guide including: description and objective of the 
simulation task; case text; instructions for accessing and starting the simulator on the computer network; 
instructions for running the simulator; sheets for strategy map review. The decisions made on the simulation 
and its results were automatically stored in a protected spreadsheet on the participant’s computer. 



Figure 1 . Experimental procedure 

The firm was run by using the balanced scorecard and the strategy map. They analyzed business status 
using the simulator interface, used this information to review the strategy and objectives and decision 
making, and then repeated the process. The experiment procedure involved four sessions and had the 
following steps (Figure 1). Session 1: a) The participants answered a questionnaire on BSC approach. This 
questionnaire, comprising twenty five multiple choice questions about the BSC, and a ten-point scale to 
evaluate the relative relevance of ten critical concepts (five regarding the performance measurement system, 
and other five regarding the systems and strategic learning perspective), captured their initial level of 
comprehension of the BSC approach. Session 2: b) The participants read the introduction with the overall 
description and the objectives of the simulation and the business case and were instructed to raise any 
questions they had as they proceeded with the case, c) The participants read the instructions for accessing, 
starting and running the simulator and they were given oral instructions with examples to show simulator 
operation, d) A first simulation was conducted to familiarize participants with the game interfaces and 
commands, and at this point they were instructed to ask for help at any time. Session 3: e) The participants 
received instruction in how to draw the strategy map, and how to review it by cutting or inserting links 
between the indicators and defining the arrows that indicated the cause-and-effect relationships, f) The 
participants drew the initial strategy map. g) The participants performed the definitive simulation. They were 
also asked to review the causal diagram (strategy map). They cut or inserted links so that the causal diagram 
accurately expressed their latest understanding of the simulated business system. The participants were also 
encouraged to use the strategy map to reflect on strategy, objectives, and decisions. Session 4: h) The 
participants filled out the same questionnaire that was used on the session 1 in order to capture their ultimate 
understanding of the certain critical concepts of BSC approach. Finally, the participants described their 
simulation experience in a written report which was part of the evaluation process. 

3.1 Research Variables 

Level of understanding of the BSC performance measurement system (LUPMS-BS and LUPMS-AS). These 
variables were measured in terms of the evaluation score of participants in answering a questionnaire, before 
the simulation experiment (LUPMS-BS) and after the simulation experiment (LUPMS-AS). 

Level of understanding of the BSC as a strategic learning system (LUSLS-BS and LUSLS-AS). These 
variables were measured in terms of the evaluation score of participants in answering a questionnaire before 
the simulation experiment (LUSLS-BS) and after the simulation experiment (LUSLS-AS). 

Level of relevance of the performance measurement system concepts (LRPMS-BS, LRPMS-AS and 
LRPMS-CA). These variables represent the students’ perception on the relative importance of the 
performance measurement system of the BSC approach. LRPMS-BS and LRPMS-AS were obtained from 
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the questionnaire answered by participants, before and after the simulation experiment. The variable 
LRPMS-CA results from a quantitative content analysis of the final report. 

Level of relevance of the strategic learning system concepts (LRSLS-BS, LRSLS-AS and LRSLS-CA). 
These variables represent the students’ perception on the relative importance of the systems and strategic 
learning perspective of the BSC approach. They also result from the questionnaire answered by participants, 
before and after the simulation experiment. The variable LRSLS-CA was also obtained from quantitative 
content analysis of the final report. 

Performance - Task performance was measured by total financial value creation. 


4. RESULTS AND DISCUSSION 

Table 1 shows the results of statistical testing to identify differences in means between stages. Unexpectedly, 
after simulation the participants showed on average a lower level of understanding of the BSC performance 
measurement system (LUPMS). However, the mean values of LUPMS for after simulation stage were not 
significantly different from the equivalent values for before simulation stage (mean difference=-0.32, p=315). 
On average, after simulation, the participants showed a better level of understanding of the BSC as a strategic 
learning system (LUSLS), a lower level of relevance of the performance measurement system concepts 
(LRPMS), and a better level of relevance of the strategic learning system concepts (LRSLS). Table 1 shows 
that those differences are significant at p<0.01 (mean difference LUSLS=0.188, p<0.001; mean difference 
LRPMS=-3.39, p<0.001; LRSLS=3.63, p=0.001). 


Table 1. Tests of significance for differences in means between the stages 


Variable 


After Simulation - Before Simulation 


Mean 

Difference 

Standard Deviation 

Significance 

P 

LUPMS 

-0.32 

2.44 

0.315 

LUSLS 

1.88* 

2.85 

0.000 

LRPMS 

-3.39* 

6.96 

0.000 

LRSLS 

3.63* 

7.65 

0.001 


*p<0.01 


Table 2. Tests of significance for differences in means for BSC concepts between stages 



After Simulation - Before Simulation 

BSC Concept 

Mean 

Difference 

Standard Deviation 

Significance 

P 

Concepts included in LRPMS 
Cl- BSC perspectives 

-2.20** 

3.49 

0.000 

C 2 - Financial measures 

-0.85* 

2.96 

0.032 

C 3 - Non-financial measures 

-0.03 

3.30 

0.937 

C 4 - Objectives and targets 

-0.25 

3.42 

0.571 

C 5 - Key initiatives 

-0.05 

2.96 

0.895 

Concepts included in LRSLS 
C 6 - Systems and dynamical view 

0.49 

4.20 

0.372 

C 7 - BSC strategy map 

LOO* 

'ill 

0.046 

C 8 - Understanding and validation of the cause- 

1 47 ** 

2.87 

0.000 

and-effect relationships 

C 9 - Inference about future performance 

0.22 

3.62 

0.642 

CIO - Double-loop learning process 

0.44 

3.80 

0.376 


*p<0.05; **p<0.01 


Table 2 shows the results of statistical testing to identify differences in means between the stage before 
simulation and the stage after simulation for each BSC concept. The lower mean value of LRPMS revealed 
by participants after simulation was mainly due to Cl - BSC perspectives (mean difference =-2.20, p<0.001) 
and C2 - Financial measures (mean difference =-0.85, p<0.05) concepts. These results suggest that after 
simulation the participants changed their perception about the importance of those concepts as they rated on 
average a lower level. Results in table 2 also suggest that the better mean value of LRSLS after simulation 
was mainly due to C7 - BSC strategy map (mean difference =1.00, p<0.05) and C8 - Understanding and 
validation of the cause-and-effect relationships (mean difference =1.47, p<0.001 ) concepts. Thus, on 
opposite, it seems that the simulation experience made the participants changing their view on the relevance 
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of these concepts as they rated on average a higher level. The relevance of those BSC concepts was also 
evaluated by means of a quantitative content analysis of the final report (table 3). According to that analysis, 
the participants perceive on average that C7 (BSC strategy map), C8 (understanding and validation of the 
cause-and-effect relationships) and CIO (Double-loop learning process) are the most relevant BSC concepts. 

In order to evaluate the relationship between Performance and each variable, a bivariate correlation 
analysis was conducted. As it can be seen in table 4, there is some correlation between Performance and 
LUPMS-AS (Corr=0.245, p=0.053). However, that correlation is not significant (at p<0.05). Performance is 
significant correlated (at p<0.05) with LUSLS-AS, and with LRSLS-CA. 


Table 3. Results of quantitative content analysis of the simulation reports 


BSC Concept 

Quantitative Content Analysis of 
Simulation Reports 
Number of 

Mentions % 

Concepts included in LRPMS 

63 

36.0% 

C 1 - BSC perspectives 

16 

9.1% 

C 2 - Financial measures 

7 

4.0% 

C 3 ' Non-fmancial measures 

18 

10.3% 

C 4 - Objectives and targets 

7 

4.0% 

C 5 - Key initiatives 

15 

8.6% 

Concepts included in LRSLS 

112 

64.0% 

C 6 - Systems and dynamical view 

6 

3.4% 

C 7 - BSC strategy map 

31 

17.7% 

C 8 - Understanding and validation of the cause- 
and-effect relationships 

32 

18.3% 

C 9 - Inference about future performance 

19 

10.9% 

CIO - Double-loop learning process 

24 

13.7% 


Table 4. Analysis of correlation (Pearson) 


Variables 

Correlation Analysis 

Performance 


Pearson Correlation 

Significance 

LUPMS-AS 

0.245 

0.053 

LUSLS-AS 

0.298* 

0.018 

LRPMS-AS 

0.019 

0.882 

LRSLS-AS 

-0.030 

0.817 

LRPMS-CA 

0.186 

0.144 

LRSLS-CA 

0.299* 

0.017 


*p<0.05 


Table 5 shows the results of multivariate regression analyses of Performance on the independent variables 
(model 1). The regression model was then refined (model 5) by performing a stepwise regression in order to 
exclude the variables that did not seem to significantly explain the dependent variable and to preserve the 
most significant explanatory variables. Regression analysis of Performance on the most significant 
independent variables shows significant effects for LUPMS-AS and LRSLS-CA. 

These results confirm five of the six hypotheses (table 6). The present study does not support Hypothesis 
HI as from table 1 the mean values of LUPMS for before and after simulation stages were not significantly 
different. On average, the participants in after simulation stage showed better LUSLS and LRSLS, with 
significant differences evident in Table 1, supporting Hypothesis H2 and Hypothesis H3. The correlation 
analysis (table 4) showed no significant relationship between LUPMS-AS and Performance. However, the 
results of multivariate regression analysis of Performance on the independent variables (model 5 - table 5) 
indicates a significant effect for LUPMS-AS. Consequently, the results provide support of Hypothesis H4. 
The results from correlation analysis (table 4) provide support of Hypothesis H5 as it was found a significant 
correlation between LUSLS-AS and Performance. LRSLS was measured through two processes: data 
obtained from the questionnaires answered by participants after simulation (variable LRSLS-AS) and data 
extracted from quantitative content analysis of participants’ simulation report (variable LRSLS-CA). 
Correlation between LRSLS-AS and Performance is not significant. However, results reveal a significant 
correlation between LRSLS-CA and Performance. Thus, the results provide support of Hypothesis H6. 
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Table 5. Model 1: regression results for all independent variables; Model 5: regression results obtained through a 

stepwise procedure 




Dependent Variable 

Independent 

Variables 


Performance 

Standardized Beta Significance 

Model 1 
LUPMS - AS 

0.226 

0.088 

LUSLS - AS 

0.144 

0.273 

LRPMS - AS 

0.111 

0.783 

LRSLS - AS 

0.053 

0.894 

LRPMS - CA 

0.176 

0.151 

LRSLS - CA 

0.336** 

0.009 

Model 5 
LUPMS - AS 

0.292* 

0.016 

LRSLS - CA 

0.340** 

0.006 


*p<0.05; **p<0.01 


Table 6. Summary of Hypothesis Testing 


Hypotheses 

Description 

Results 

HI 

The use of simulation improves the level of understanding of the BSC 
performance measurement system 

Not supported 

H2 

The use of simulation improves the level of understanding of the BSC as 
a strategic learning system 

Supported 

H3 

The use of simulation improves the level of perceived relevance of the 
strategic learning system concepts 

Supported 

H4 

The level of understanding of the BSC performance measurement system 
is positively correlated with Performance 

Supported 

H5 

The level of understanding of the BSC as a strategic learning system is 
positively correlated with Performance 

Supported 

H6 

The level of perceived relevance of the strategic learning system concepts 
is positively correlated with Performance 

Supported 


4. CONCLUSION 

Kaplan and Norton introduced the Balanced Scorecard (BSC) which is based on a systems perspective of the 
business strategy and performance measurement. Many organizations around the world are using the BSC to 
define, implement and manage strategy. Nevertheless there exist studies that show in general terms that 
managers do not understand the BSC as the measures and perspectives in use are fairly independent, and do 
not always mirror the recommended cause-and-effect logic included in the systems perspective of the BSC 
approach. Thus, appropriate teaching and training methodologies must be designed in order to improve 
subjects learning and comprehension of those concepts. The present study addresses the effectiveness of 
teaching the Balanced Scorecard by means of business simulation. An experiment that uses a business 
simulator is performed for testing a set of hypotheses about the influence of simulation on the subjects’ 
understanding of the BSC. The simulation methodology significantly enhanced the understanding of the BSC 
concepts related to the strategic management and double-loop learning processes and the systems 
perspective. Results also suggested that understanding of those BSC concepts positively influences 
simulation performance. 

The findings confirm our assumptions on the effectiveness of using simulation for teaching the BSC 
approach as a strategic management system (involving balanced scorecards and strategy maps as suggested 
by Kaplan and Norton). The simulator operates by participants developing and combining critical resources 
in appropriate levels in order to attract and retain customers, while running an efficient company. To reach 
this goal, the participants must understand the interdependence among critical resources and variables of the 
business case and they must combine these effectively. As students create a causal model (the BSC strategy 
map) representing the critical cause-and-effect relations of the business case, and use that causal model and 
the feedback information from the BSC performance system for running the firm, they develop an effective 
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understanding about the meaningless and usefulness of the BSC concepts related to the strategic management 
and double-loop learning processes. 
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